Information processing method for displaying a virtual screen and system for executing the information processing method

ABSTRACT

A method includes generating first virtual space data for defining a first virtual space. The first virtual space includes a first avatar associated with a first user terminal; and a first virtual room including the first avatar and a first virtual screen. The method includes detecting a movement of a head mounted device (HMD) included in the first user terminal. The method includes identifying a visual field in accordance with the detected movement of the HMD. The method includes displaying, on the HMD, a visual-field image corresponding to the visual field. The method includes receiving, from a second user terminal, a visiting request signal requesting that a second avatar associated with the second user terminal visit the first virtual room. The method includes updating the first virtual space data by updating a size of the first virtual screen based on a number of avatars arranged in the first virtual room.

RELATED APPLICATIONS

The present application claims priority to JP2016-238745 filed Dec. 8,2016 and JP2016-255987 filed Dec. 28, 2016, the disclosures of which arehereby incorporated by reference herein in their entirety.

BACKGROUND

This disclosure relates to an information processing method and a systemfor executing the information processing method.

In Japanese Patent Application Laid-open No. 2014-017776, there isdescribed a technology for identifying a positional relationship among aplurality of users in the real space and then determining a positionalrelationship between a plurality of user characters (avatars) within avirtual space based on the identified positional relationship among theplurality of users.

In WO 2016/002445 A1, there is described an information processingsystem in which only users having lines of sight directed to the sameobject among a plurality of users sharing content are allowed toexchange voice information with one another.

In U.S. Pat. No. 9,396,588, there is described a technology forarranging a virtual screen, a virtual video projector, and virtualseats, which are provided as a virtual movie theater, within a virtualspace, to thereby provide a user wearing a head-mounted device(hereinafter referred to simply as “HMD”) with an experience of enjoyinga movie at a movie theater.

In Japanese Patent Application Laid-open No. 2014-017776, WO 2016/002445A1, and U.S. Pat. No. 9,396,588, there is no description regarding howto control a virtual room in which a plurality of user characters(avatars) are arranged.

SUMMARY

(1) According to at least one embodiment of this disclosure, there isprovided an information processing method to be executed by a computerin a virtual space distribution system. The virtual space distributionsystem includes a plurality of user terminals each including ahead-mounted display to be worn on a head of a user; and a server.

The information processing method includes generating first virtualspace data for defining a first virtual space. The first virtual spaceincludes a first avatar associated with a first user terminal among theplurality of user terminals; and a first virtual room in which the firstavatar is arranged. The method further includes receiving, from a seconduser terminal among the plurality of user terminals, a visiting requestsignal for requesting that a second avatar associated with the seconduser terminal visit the first virtual room. The method further includesdetermining, in response to the visit request signal, whether or not thefirst virtual room is to be updated based on a total number of avatarscurrently arranged in the first virtual room and second avatars. Themethod further includes updating, in response to a determination thatthe first virtual room is to be updated, the first virtual room so as toupdate a size of the first virtual room.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a virtual space distribution systemaccording to at least one embodiment of this disclosure.

FIG. 2 is a schematic diagram of a user terminal according to at leastone embodiment of this disclosure.

FIG. 3 is a diagram of a head of a user wearing an HMD according to atleast one embodiment of this disclosure.

FIG. 4 is a diagram of a hardware configuration of a control deviceaccording to at least one embodiment of this disclosure.

FIG. 5 is a flowchart of processing for displaying a visual-field imageon the HMD according to at least one embodiment of this disclosure.

FIG. 6 is an xyz spatial diagram of a virtual space according to atleast one embodiment of this disclosure.

FIG. 7A is a yx plane diagram of the virtual space illustrated in FIG.6.

FIG. 7B is a zx plane diagram of the virtual space according to at leastone embodiment of this disclosure.

FIG. 8 is a diagram of the visual-field image displayed on the HMDaccording to at least one embodiment of this disclosure.

FIG. 9 is a diagram of a hardware configuration of a server according toat least one embodiment of this disclosure.

FIG. 10A is a diagram of the virtual space provided to a user Aaccording to at least one embodiment of this disclosure.

FIG. 10B is a diagram of the virtual space provided to a user Baccording to at least one embodiment of this disclosure.

FIG. 11 is a sequence diagram of processing for synchronizing movementsof respective avatars between user terminals according to at least oneembodiment of this disclosure.

FIG. 12A is a sequence diagram of an information processing methodaccording to at least one embodiment.

FIG. 12B is a sequence diagram of the information processing methodaccording to at least one embodiment continued from FIG. 12A.

FIG. 13A is a diagram of the virtual space provided to the user Aaccording to at least one embodiment of this disclosure.

FIG. 13B is a diagram of the virtual space provided to a user Caccording to at least one embodiment of this disclosure.

FIG. 14A is a diagram of a virtual room of an avatar of the user A,which has been enlarged according to at least one embodiment of thisdisclosure.

FIG. 14B is a diagram of the virtual space of an avatar of the user C,which has been updated according to at least one embodiment of thisdisclosure.

FIG. 15 is a sequence diagram of processing for viewing video contentaccording to at least one embodiment of this disclosure.

FIG. 16A is a table of a video content charge table according to atleast one embodiment of this disclosure.

FIG. 16B is a table of a billing table according to at least oneembodiment of this disclosure.

FIG. 17 is a sequence diagram of processing for enlarging a virtualscreen according to at least one embodiment of this disclosure.

FIG. 18A is a diagram of the virtual space provided to the user Aaccording to at least one embodiment of this disclosure.

FIG. 18B is a diagram of the virtual space provided to the user Aaccording to at least one embodiment of this disclosure.

FIG. 19A is a diagram of the virtual space provided to the user Aaccording to at least one embodiment of this disclosure.

FIG. 19B is a diagram of the virtual space provided to the user Baccording to at least one embodiment of this disclosure.

FIG. 20 is a sequence diagram of the processing for synchronizing themovements of the respective avatars between the user terminals accordingto at least one embodiment of this disclosure.

FIG. 21 is a flowchart of an information processing method according toat least one embodiment of this disclosure.

FIG. 22 is a plan view of the virtual space provided to the user Aaccording to at least one embodiment of this disclosure.

FIG. 23 is a diagram of the visual-field image displayed on the HMD of auser terminal 1A according to at least one embodiment of thisdisclosure.

FIG. 24 is a diagram of the visual-field image displayed on the HMD ofthe user terminal 1A according to at least one embodiment of thisdisclosure.

FIG. 25 is a flowchart of an information processing method according toat least one embodiment of this disclosure.

FIG. 26 is a diagram of the virtual space provided to the user Aaccording to at least one embodiment of this disclosure.

FIG. 27 is a diagram of the visual-field image displayed on the HMD ofthe user terminal 1A according to at least one embodiment of thisdisclosure.

FIG. 28 is a flowchart of an information processing method according toat least one embodiment of this disclosure.

FIG. 29A is a diagram of the virtual space provided to the user Aaccording to at least one embodiment of this disclosure.

FIG. 29B is a diagram of the virtual space provided to the user A, whichindicates a state under which an avatar 4A has turned to face an avatar4B according to at least one embodiment of this disclosure.

FIG. 30A is a diagram of a state of a face of the avatar 4B exhibited inFIG. 29A according to at least one embodiment of this disclosure.

FIG. 30B is a diagram of the state of the face of the avatar 4Bexhibited in FIG. 29B according to at least one embodiment of thisdisclosure.

DETAILED DESCRIPTION

Embodiments of this disclosure are described below with reference to thedrawings. Once a component is described in this description of theembodiments, a description on a component having the same referencenumber as that of the already described component is omitted for thesake of convenience.

With reference to FIG. 1, a description is given of a schematicconfiguration of a virtual space distribution system 100 (or a social VRsystem). FIG. 1 is a schematic diagram of the virtual space distributionsystem 100 (hereinafter referred to simply as “distribution system 100”)according to at least one embodiment of this disclosure. In FIG. 1, thedistribution system 100 includes a user terminal 1A operated by a userA, a user terminal 1B operated by a user B, a user terminal 1C operatedby a user C, and a server 2. The user terminals 1A to 1C are connectedto the server 2 through a communication network 3, for example, theInternet, so as to enable communication between each of the userterminals 1A to 1C and the server 2. In at least one embodiment, avirtual space includes a virtual reality (VR) space, an augmentedreality (AR) space, or a mixed reality (MR) space. In the following, forthe sake of convenience in description, each of the user terminals 1A to1C may be collectively referred to simply as “user terminal 1”. Further,each of the users A to C may be collectively referred to simply as “userU”. In at least one embodiment, the user terminals 1A to 1C are assumedto have the same configuration.

With reference to FIG. 2, a description is given of a configuration ofthe user terminal 1. FIG. 2 is a schematic diagram of the user terminal1 according to at least one embodiment of this disclosure. In FIG. 2,the user terminal 1 includes a head-mounted device (HMD) 110 worn on ahead of the user U, headphones 116, a microphone 118, a position sensor130, an external controller 320, and a control device 120.

The HMD 110 includes a display unit 112, an HMD sensor 114, and an eyegaze sensor 140. The display unit 112 includes a non-transmissivedisplay device configured to completely cover a field of view (visualfield) of the user U wearing the HMD 110. With this, the user U can seeonly a visual-field image displayed on the display unit 112, and hencethe user U can be immersed in a virtual space. The display unit 112 mayinclude a left-eye display unit configured to provide an image to a lefteye of the user U, and a right-eye display unit configured to provide animage to a right eye of the user U. The HMD 110 may include atransmissive display device. In this case, the transmissive displaydevice may be configured to temporarily function as a non-transmissivedisplay device through adjustment of a transmittance.

The HMD sensor 114 is mounted near the display unit 112 of the HMD 110.The HMD sensor 114 includes at least one of a geomagnetic sensor, anacceleration sensor, and an inclination sensor (for example, an angularvelocity sensor or a gyro sensor), and can detect various movements ofthe HMD 110 worn on the head of the user U.

The eye gaze sensor 140 has an eye tracking function of detecting aline-of-sight direction of the user U. For example, the eye gaze sensor140 may include a right-eye gaze sensor and a left-eye gaze sensor. Theright-eye gaze sensor may be configured to detect reflective lightreflected from the right eye (in particular, the cornea or the iris) ofthe user U by irradiating the right eye with, for example, infraredlight, to thereby acquire information relating to a rotational angle ofa right eyeball. The left-eye gaze sensor may be configured to detectreflective light reflected from the left eye (in particular, the corneaor the iris) of the user U by irradiating the left eye with, forexample, infrared light, to thereby acquire information relating to arotational angle of a left eyeball.

The headphones 116 are worn on right and left ears of the user U. Theheadphones 116 are configured to receive sound data (electrical signal)from the control device 120 to output sounds based on the received sounddata. The microphone 118 is configured to collect sounds uttered by theuser U, and to generate sound data (i.e., electric signal) based on thecollected sounds. The microphone 118 is also configured to transmit thesound data to the control device 120.

The position sensor 130 is constructed of, for example, a positiontracking camera, and is configured to detect the positions of the HMD110 and the external controller 320. The position sensor 130 isconnected to the control device 120 so as to enable communicationto/from the control device 120 in a wireless or wired manner. Theposition sensor 130 is configured to detect information relating topositions, inclinations, or light emitting intensities of a plurality ofdetection points (not shown) provided in the HMD 110. Further, theposition sensor 130 is configured to detect information relating topositions, inclinations, and/or light emitting intensities of aplurality of detection points (not shown) provided in the externalcontroller 320. The detection points are, for example, light emittingportions configured to emit infrared light or visible light. Theposition sensor 130 may include an infrared sensor or a plurality ofoptical cameras.

The external controller 320 is used to control, for example, a movementof a finger object to be displayed in the virtual space. The externalcontroller 320 may include a right-hand external controller to be usedby being held by a right hand of the user U, and a left-hand externalcontroller to be used by being held by a left hand of the user U.

The control device 120 is capable of acquiring information on theposition of the HMD 110 based on the information acquired from theposition sensor 130, and accurately associating the position of thevirtual camera (or an avatar associated with the user U) in the virtualspace with the position of the user U wearing the HMD 110 in the realspace based on the acquired information on the position of the HMD 110.Further, the control device 120 is capable of acquiring information onthe movement of the external controller 320 based on the informationacquired from the position sensor 130, and accurately associating themovement of the finger object (the fingers of the avatar associated withthe user U) to be displayed in the virtual space with the movement ofthe external controller 320 in the real space based on the acquiredinformation on the movement of the external controller 320.

The control device 120 is capable of identifying each of the line ofsight of the right eye and the line of sight of the left eye of the userU based on the information transmitted from the eye gaze sensor 140, tothereby identify a point of gaze being an intersection between the lineof sight of the right eye and the line of sight of the left eye.Further, the control device 120 is capable of identifying aline-of-sight direction of the user U based on the identified point ofgaze. In this case, the line-of-sight direction of the user U is aline-of-sight direction of both eyes of the user U, and matches adirection of a straight line passing through the point of gaze and amidpoint of a line segment connecting between the right eye and the lefteye of the user U. The control device 120 is capable of identifying adirection of an iris and pupil of the avatar associated with the user Ubased on the line-of-sight direction of the user U.

With reference to FIG. 3, a method of acquiring information relating toa position and an inclination of the HMD 110 is described. FIG. 3 is adiagram of the head of the user U wearing the HMD 110 according to atleast one embodiment of this disclosure. The information relating to theposition and the inclination of the HMD 110, which move in associationwith the movement of the head of the user U wearing the HMD 110, can bedetected by the position sensor 130 and/or the HMD sensor 114 mounted onthe HMD 110. In FIG. 3, three-dimensional coordinates (uvw coordinates)are defined about the head of the user U wearing the HMD 110. Aperpendicular direction in which the user U stands upright is defined asa v axis, a direction being orthogonal to the v axis and passing throughthe center of the HMD 110 is defined as a w axis, and a directionorthogonal to the v axis and the w axis is defined as a u direction. Theposition sensor 130 and/or the HMD sensor 114 are/is configured todetect angles about the respective uvw axes (that is, inclinationsdetermined by a yaw angle representing the rotation about the v axis, apitch angle representing the rotation about the u axis, and a roll anglerepresenting the rotation about the w axis). The control device 120 isconfigured to determine angular information for controlling a visualaxis of the virtual camera based on the detected change in angles aboutthe respective uvw axes.

With reference to FIG. 4, a hardware configuration of the control device120 is described. FIG. 4 is a diagram of the hardware configuration ofthe control device 120 according to at least one embodiment. In FIG. 4,the control device 120 includes a control unit 121, a storage unit 123,an input/output (I/O) interface 124, a communication interface 125, anda bus 126. The control unit 121, the storage unit 123, the I/O interface124, and the communication interface 125 are connected to each other viathe bus 126 so as to enable communication therebetween.

The control device 120 may be constructed as a personal computer, atablet computer, or a wearable device separately from the HMD 110, ormay be built into the HMD 110. A part of the functions of the controldevice 120 may be mounted to the HMD 110, and the remaining functions ofthe control device 120 may be mounted to another device separate fromthe HMD 110.

The control unit 121 includes a memory and a processor. The memory isconstructed of, for example, a read only memory (ROM) having variousprograms and the like stored therein or a random access memory (RAM)having a plurality of work areas in which various programs to beexecuted by the processor are stored. The processor is constructed of,for example, a central processing unit (CPU), a micro processing unit(MPU) and/or a graphics processing unit (GPU), and is configured toload, on the RAM, programs designated by various programs installed intothe ROM to execute various types of processing in cooperation with theRAM.

In particular, the control unit 121 may control various operations ofthe control device 120 by causing the processor to load a controlprogram on the RAM to execute the control program in cooperation withthe RAM. The control unit 121 displays the visual-field image on thedisplay unit 112 of the HMD 110 based on visual-field image data. Thisallows the user U to be immersed in the virtual space.

The storage unit (storage) 123 is a storage device, for example, a harddisk drive (HDD), a solid state drive (SSD), or a USB flash memory, andis configured to store programs and various types of data. The storageunit 123 may store a control program for causing a computer to executeat least a part of an information processing method according to atleast one embodiment, or a control program for achieving a social VR.The storage unit 123 may store programs for authentication of the user Uand data relating to various images and objects (for example, virtualrooms and avatars). Further, a database including tables for managingvarious types of data may be constructed in the storage unit 123.

The I/O interface 124 is configured to connect each of the positionsensor 130, the HMD 110, the external controller 320, the headphones116, and the microphone 118 to the control device 120 so as to enablecommunication therebetween, and is constructed of, for example, auniversal serial bus (USB) terminal, a digital visual interface (DVI)terminal, or a high-definition multimedia Interface® (HDMI) terminal.The control device 120 may be wirelessly connected to each of theposition sensor 130, the HMD 110, the external controller 320, theheadphones 116, and the microphone 118.

The communication interface 125 is configured to connect the controldevice 120 to the communication network 3, for example, a local areanetwork (LAN), a wide area network (WAN), or the Internet. Thecommunication interface 125 includes various wire connection terminalsand various processing circuits for wireless connection forcommunication to/from an external device, for example, the server 2, viathe communication network 3, and is configured to become compatible withcommunication standards for communication via the communication network3.

With reference to FIG. 5 to FIG. 8, processing of displaying thevisual-field image on the HMD 110 is described. FIG. 5 is a flowchart ofthe processing of displaying the visual-field image on the HMD 110according to at least one embodiment. FIG. 6 is an xyz spatial diagramof a virtual space 200 according to at least one embodiment. FIG. 7A isa yx plane diagram of the virtual space 200 in FIG. 6 according to atleast one embodiment. FIG. 7B is a zx plane diagram of the virtual space200 in FIG. 6 according to at least one embodiment. FIG. 8 is a diagramof a visual-field image V displayed on the HMD 110 according to at leastone embodiment.

In FIG. 5, in Step S1, the control unit 121 (refer to FIG. 4) generatesvirtual space data representing the virtual space 200 including avirtual camera 300 and various objects. In FIG. 6, the virtual space 200is defined as an entire celestial sphere having a center position 210 asthe center (in FIG. 6, only the upper-half celestial sphere is includedfor clarity). In the virtual space 200, an xyz coordinate system havingthe center position 210 as the origin is set. The virtual camera 300defines a visual axis L for identifying the visual-field image V (referto FIG. 8) to be displayed on the HMD 110. The uvw coordinate systemthat defines the visual field of the virtual camera 300 is determined soas to move in association with the uvw coordinate system that is definedabout the head of the user U in the real space. The control unit 121 maymove the virtual camera 300 in the virtual space 200 in association withthe movement in the real space of the user U wearing the HMD 110.

In Step S2, the control unit 121 identifies a visual field CV (refer toFIG. 7A and FIG. 7B) of the virtual camera 300. Specifically, thecontrol unit 121 acquires information relating to a position and aninclination of the HMD 110 based on data representing the state of theHMD 110, which is transmitted from the position sensor 130 and/or theHMD sensor 114. The control unit 121 identifies the position and thedirection of the virtual camera 300 in the virtual space 200 based onthe information relating to the position and the inclination of the HMD110. The control unit 121 determines the visual axis L of the virtualcamera 300 based on the position and the direction of the virtual camera300, and identifies the visual field CV of the virtual camera 300 basedon the determined visual axis L. In this case, the visual field CV ofthe virtual camera 300 corresponds to a part of the region of thevirtual space 200 that can be visually recognized by the user U wearingthe HMD 110 (in other words, corresponds to a part of the region of thevirtual space 200 to be displayed on the HMD 110). The visual field CVhas a first region CVa set as an angular range of a polar angle α aboutthe visual axis L in the xy plane in FIG. 7A, and a second region CVbset as an angular range of an azimuth β about the visual axis L in thexz plane in FIG. 7B. The control unit 121 may identify the line-of-sightdirection of the user U based on data representing the line-of-sightdirection of the user U, which is transmitted from the eye gaze sensor140, and may determine the direction of the virtual camera 300 based onthe line-of-sight direction of the user U.

The control unit 121 can identify the visual field CV of the virtualcamera 300 based on the data transmitted from the position sensor 130and/or the HMD sensor 114. In this case, when the user U wearing the HMD110 moves, the control unit 121 can change the visual field CV of thevirtual camera 300 based on the data representing the movement of theHMD 110, which is transmitted from the position sensor 130 and/or theHMD sensor 114. That is, the control unit 121 can change the visualfield CV in accordance with the movement of the HMD 110. Similarly, whenthe line-of-sight direction of the user U changes, the control unit 121may move the visual field CV of the virtual camera 300 based on the datarepresenting the line-of-sight direction of the user U, which istransmitted from the eye gaze sensor 140. That is, the control unit 121may change the visual field CV in accordance with the change in theline-of-sight direction of the user U.

In Step S3, the control unit 121 generates visual-field image datarepresenting the visual-field image V to be displayed on the displayunit 112 of the HMD 110. Specifically, the control unit 121 generatesthe visual-field image data based on the virtual space data for definingthe virtual space 200 and the visual field CV of the virtual camera 300.

In Step S4, the control unit 121 displays the visual-field image V onthe display unit 112 of the HMD 110 based on the visual-field image data(refer to FIG. 7A and FIG. 7B). The visual field CV of the virtualcamera 300 changes in accordance with the movement of the user U wearingthe HMD 110, and hence the visual-field image V to be displayed on thedisplay unit 112 of the HMD 110 changes as well. Thus, the user U can beimmersed in the virtual space 200.

The virtual camera 300 may include a left-eye virtual camera and aright-eye virtual camera. In this case, the control unit 121 generatesleft-eye visual-field image data representing a left-eye visual-fieldimage based on the virtual space data and the visual field of theleft-eye virtual camera. Further, the control unit 121 generatesright-eye visual-field image data representing a right-eye visual-fieldimage based on the virtual space data and the visual field of theright-eye virtual camera. The control unit 121 displays the left-eyevisual-field image on a left-eye display unit based on the left-eyevisual-field image data, and displays the right-eye visual-field imageon a right-eye display unit based on the right-eye visual-field imagedata. In this manner, the user U can visually recognize the visual-fieldimage three-dimensionally owing to parallax between the left-eyevisual-field image and the right-eye visual-field image. The virtualcamera may be arranged at a position of the eye of the avatar operatedby the user as described later. For example, the left-eye virtual cameramay be arranged at the left eye of the avatar, while the right-eyevirtual camera may be arranged at the right eye of the avatar.

A hardware configuration of the server 2 in FIG. 1 is described withreference to FIG. 9. FIG. 9 is a diagram of the hardware configurationof the server 2 according to at least one embodiment. In FIG. 9, theserver 2 includes a control unit 23, a storage unit 22, a communicationinterface 21, and a bus 24. The control unit 23, the storage unit 22,and the communication interface 21 are connected to one another throughthe bus 24 so as to enable communication therebetween. The control unit23 includes a memory and a processor. The memory is formed of, forexample, a ROM and a RAM, and the processor is formed of, for example, aCPU, an MPU and/or a GPU.

The storage unit (storage) 22 is, for example, a large capacity HDD. Thestorage unit 22 may store a control program for causing a computer toexecute at least a part of the information processing method accordingto at least one embodiment, or a control program for achieving a socialVR. The storage unit 22 may store user management information formanaging each user, various images, and data relating to objects (forexample, a virtual room and an avatar). The storage unit 22 may alsostore a video content charge table and a billing table, which are shownin FIG. 16, as databases. The communication interface 21 is configuredto connect the server 2 to the communication network 3.

With reference to FIG. 1, FIG. 10, and FIG. 11, a description is givenof processing for synchronizing movements of respective avatars 4A and4B between the user terminal 1A and the user terminal 1B according to atleast one embodiment. FIG. 10A is a diagram of a virtual space 200Aprovided to the user A according to at least one embodiment. FIG. 10B isa diagram of a virtual space 200B provided to the user B according to atleast one embodiment. FIG. 11 is a sequence diagram of the processingfor synchronizing the movements of the respective avatars 4A and 4Bbetween the user terminal 1A and the user terminal 1B according to atleast one embodiment. In this description, the avatar 4A associated withthe user terminal 1A (user A) and the avatar 4B associated with the userterminal 1B (user B) are arranged in a virtual room 7A of the avatar 4A.That is, the user A and the user B share one virtual room 7A via thenetwork 3 after the user B visits the virtual room 7A of the user A. Oneof ordinary skill in the art would understand that variations arepossible, such as user A visiting the virtual room of user B, multipleusers visiting the virtual room 7A of user A, etc.

In FIG. 10A, the virtual space 200A of the user A includes the avatar4A, the avatar 4B, and the virtual room 7A in which the avatars 4A and4B are arranged. The avatar 4A is operated by the user A, and isconfigured to move in association with the action of the user A. Theavatar 4B is operated by the user B, and is configured to move inassociation with the action of the user B. For example, a position ofthe avatar 4A (4B) may be identified based on a position of the HMD 110of the user A (user B). In the same manner, a direction of a face of theavatar 4A (4B) may be identified based on an inclination of the HMD 110of the user A (user B). In addition, an action of a hand of the avatar4A (4B) may be identified based on the actions of the externalcontrollers. A line of sight (for example, a position of the iris andpupil) of the avatar 4A (4B) may be identified based on theline-of-sight direction of the user A (user B) detected by the eye gazesensor 140. The virtual camera 300 in FIG. 6 may be arranged at the eyeof the avatar 4A (4B). Specifically, the left-eye virtual camera may bearranged at the left eye of the avatar 4A (4B), while the right-eyevirtual camera may be arranged at the right eye of the avatar 4A (4B).In the following description, it is assumed that the virtual camera 300is arranged at the eyes of the avatar 4A (4B).

The virtual room 7A includes a virtual table 5A and a virtual screen 6Aconfigured to display video content (moving image). The user A (B) canenjoy the video content displayed on the virtual screen 6A through thevisual-field image displayed on the HMD 110. The user A may possessdifferent kinds of virtual rooms to be arranged in the virtual space200A. A plurality of virtual rooms possessed by the user A may be storedin, for example, the storage unit 123 of the user terminal 1A or thestorage unit 22 of the server 2. For example, the user A can change apredetermined virtual room arranged in the virtual space to anothervirtual room.

In FIG. 10B, the virtual space 200B of the user B includes the avatar4A, the avatar 4B, and the virtual room 7A in which the avatars 4A and4B are arranged. Before the avatar 4B visits the virtual room 7A of theavatar 4A, a virtual room of the avatar 4B is arranged in the virtualspace 200B.

With reference to FIG. 11, in Step S10, the control unit 121 of the userterminal 1A generates control information on the avatar 4A, and thentransmits the generated control information on the avatar 4A to theserver 2. The control unit 23 of the server 2 receives the controlinformation on the avatar 4A from the user terminal 1A (Step S11). Inthis case, the control information on the avatar 4A representsinformation required for controlling the action of the avatar 4A, andincludes, for example, information relating to the position of theavatar 4A, information relating to the direction of the face of theavatar 4A, information relating to the action of the hand of the avatar4A, and information relating to the line of sight of the avatar 4A.

In Step S12, the control unit 121 of the user terminal 1B generatescontrol information on the avatar 4B, and then transmits the generatedcontrol information on the avatar 4B to the server 2. The control unit23 of the server 2 receives the control information on the avatar 4Bfrom the user terminal 1B (Step S13). In this case, the controlinformation on the avatar 4B represents information required forcontrolling the action of the avatar 4B, and includes, for example,information relating to the position of the avatar 4B, informationrelating to the direction of the face of the avatar 4B, informationrelating to the action of the hand of the avatar 4B, and informationrelating to the line of sight of the avatar 4B.

The server 2 transmits the control information on the avatar 4B to theuser terminal 1A (Step S14) while transmitting the control informationon the avatar 4A to the user terminal 1B (Step S18). After receiving thecontrol information on the avatar 4B in Step S15, the control unit 121of the user terminal 1A updates the virtual space data representing thevirtual space 200A in FIG. 10A based on the control information on theavatars 4A and 4B (Step S16). For example, when the avatars 4A and 4Bmove, the virtual space data representing the virtual space includingthe avatars 4A and 4B that have moved and the virtual room 7A isgenerated. The control unit 121 of the user terminal 1A identifies thevisual field CV of the virtual camera 300, which is illustrated in FIG.7, based on the position and the inclination of the HMD 110, and thenupdates the visual-field image displayed on the HMD 110 based on theupdated virtual space data and the visual field CV of the virtual camera300 (Step S17).

After receiving the control information on the avatar 4A in Step S19,the control unit 121 of the user terminal 1B updates the virtual spacedata representing the virtual space 200B in FIG. 10B based on thecontrol information on the avatars 4A and 4B (Step S20). The controlunit 121 of the user terminal 1B identifies the visual field CV of thevirtual camera 300 based on the position and the inclination of the HMD110, and then updates the visual-field image displayed on the HMD 110based on the updated virtual space data and the visual field CV of thevirtual camera 300 (Step S21).

In at least one embodiment, after the user terminals 1A and 1B transmitthe control information on the avatar 4A and the control information onthe avatar 4B, respectively, to the server 2, the server 2 transmits thecontrol information on the avatar 4A to the user terminal 1B, andtransmits the control information on the avatar 4B to the user terminal1A. The movements of the respective avatars 4A and 4B can besynchronized between the user terminal 1A and the user terminal 1B. Therespective processing steps in FIG. 11 may be executed for every frame.

In a case of synchronizing the movements of the respective avatars 4A to4C among the user terminals 1A to 1C, the user terminal 1A transmits thecontrol information on the avatar 4A to the server 2, the user terminal1B transmits the control information on the avatar 4B to the server 2,and the user terminal 1C transmits control information on the avatar 4Cto the server 2. The server 2 transmits the control information on theavatars 4A and 4C to the user terminal 1B, transmits the controlinformation on the avatars 4B and 4C to the user terminal 1A, andtransmits the control information on the avatars 4A and 4B to the userterminal 1C.

With reference to FIG. 12A to FIG. 14, a description is given of theinformation processing method according to at least one embodiment. FIG.12A is a sequence diagram of the information processing method accordingto at least one embodiment. FIG. 12B is a sequence diagram continuingfrom the information processing method of FIG. 12A according to at leastone embodiment. FIG. 13A is a diagram of the virtual space 200A providedto the user A (before a visit of the avatar 4C) according to at leastone embodiment. FIG. 13B is a diagram of a virtual space 200C of theuser C (before the visit of the avatar 4C) according to at least oneembodiment. FIG. 14A is a diagram of the virtual room of the avatar 4A,which has been enlarged by the visit of the avatar 4C to the virtualroom of the avatar 4A according to at least one embodiment. FIG. 14B isa diagram of the virtual space of the avatar 4C, which has been updatedby the visit of the avatar 4C to the virtual room of the avatar 4Aaccording to at least one embodiment.

In this description, the avatar 4A and the avatar 4B are arranged in thevirtual room 7A of the avatar 4A. That is, the avatar 4B has visited thevirtual room 7A of the avatar 4A. The avatar 4C associated with the userterminal 1C (user C) is arranged in a virtual room 7C of the avatar 4C.The avatar 4C is to visit the virtual room 7A of the user A. One ofordinary skill in the art understand that variations are possible.

In FIG. 13A, the virtual space 200A of the user A (first virtual space)includes the avatar 4A (first avatar) associated with the user terminal1A (first user terminal), the avatar 4B associated with the userterminal 1B, and the virtual room 7A (first virtual room). As describedabove, the avatar 4A is operated by the user A (first user), and isconfigured to move in association with the action of the user A. Theavatar 4B is operated by the user B, and is configured to move inassociation with the action of the user B. The virtual room 7A includesthe virtual table 5A and the virtual screen 6A (first virtual screen). Asize of the virtual room 7A is a small (s) size. The virtual space data(first virtual space data) representing the virtual space 200A isgenerated by the control unit 121 of the user terminal 1A.

In FIG. 13B, the virtual space 200C of the user C includes the avatar 4C(second avatar) associated with the user terminal 1C (second userterminal) and the virtual room 7C where the avatar 4C is arranged. Theavatar 4C is operated by the user C (second user), and is configured tomove in association with the action of the user C. The virtual spacedata representing the virtual space 200C is generated by the controlunit 121 of the user terminal 1C.

With reference to FIG. 12A, in Step S30, the user C executespredetermined operation input for visiting the virtual room 7A of theavatar 4A. For example, the user C selects an item “Visit the virtualroom of the avatar 4A” or other such item within the menu screendisplayed on the HMD 110. The control unit 121 of the user terminal 1Cgenerates a visiting request signal for requesting that the avatar 4Cvisit the virtual room 7A of the avatar 4A based on the operation inputreceived from the user C, and then transmits the visiting request signalto the server 2.

In Step S31, the control unit 23 of the server 2 receives the visitingrequest signal from the user terminal 1C. The control unit 23 determineswhether or not to update the virtual room 7A, that is, to enlarge thevirtual room 7A, based on a total number N of avatars currently arrangedin the virtual room 7A and avatars that are to visit the virtual room 7A(Step S32). A relationship between the total number N of avatarscurrently arranged in the virtual room and avatars that are to visit thevirtual room and the size of the virtual room is, for example, asfollows.

[Table 1]

TABLE 1 Relationship between total number N of avatars and size ofvirtual room Total number N of avatars Size of virtual room 1 ≤ N < 3Small (S) size 3 ≤ N < 6 Medium (M) size 6 ≤ N Large (L) size

A table indicating a relationship between the total number N of avatarsand the size of the virtual room, which is shown in Table 1, may bestored in the storage unit 22 of the server 2. In at least oneembodiment, the number of avatars currently arranged in the virtual room7A is two (avatars 4A and 4B), while the number of avatars that are tovisit the virtual room 7A is one (avatar 4C), and hence the control unit23 identifies the total number N of avatars as three. The control unit23 determines that the size of the virtual room of the avatar 4A is tobe changed from the S size to a medium (M) size, that is, the size ofthe virtual room of the avatar 4A is to be enlarged, based on the totalnumber N of avatars being three and the table indicating therelationship between the total number N of avatars and the size of thevirtual room (YES in Step S32).

The control unit 23 acquires a virtual room 70A of the M size, in FIG.14, from the storage unit 22, and then transmits the virtual room 70A ofthe M size (enlarged first virtual room), the avatars 4A and 4B, and thecontrol information on the avatars 4A and 4B to the user terminal 1C(Step S33). Although not shown in FIG. 12, the server 2 is assumed toacquire the control information on the avatars 4A to 4C for every frame.The virtual room 7A is a virtual room of the S size of the avatar 4A,and the virtual room 70A is a virtual room of the M size of the avatar4A.

In Step S34, the control unit 121 of the user terminal 1C receives thevirtual room 70A, the avatars 4A and 4B, and the control information onthe avatars 4A and 4B. The control unit 121 updates the virtual spacedata representing the virtual space 200C in FIG. 13B based on thevirtual room 70A, the avatars 4A to 4C, and the control information onthe avatars 4A to 4C (Step S35). Specifically, the control unit 121generates the virtual space data representing the virtual space 220C inFIG. 14B. In FIG. 14B, the virtual space 220C includes the avatars 4A,4B, and 4C and the virtual room 70A. The virtual room 70A includes thevirtual table 5A and a virtual screen 60A (enlarged first virtualscreen).

The control unit 121 of the user terminal 1C identifies the visual fieldCV of the virtual camera 300, in FIG. 7, based on the position and theinclination of the HMD 110, and then updates the visual-field imagedisplayed on the HMD 110 based on the updated virtual space data and thevisual field CV of the virtual camera 300 (Step S36). When the user Cexecutes the predetermined operation input for visiting the virtual room7A of the avatar 4A, the virtual space to be presented to the user C ischanged from the virtual space 200C including the virtual room 7C, inFIG. 13B, to the virtual space 220C including the virtual room 70A ofthe avatar 4A, in FIG. 14B, and hence the avatar 4C can visit thevirtual room of the avatar 4A.

In Step S37, the control unit 23 transmits the virtual room 70A of the Msize (enlarged first virtual room), the avatar 4C, and the controlinformation on the avatars 4C and 4B to the user terminal 1A. In StepS38, the control unit 121 of the user terminal 1A receives the virtualroom 70A, the avatar 4C, and the control information on the avatars 4Cand 4B. The control unit 121 updates the virtual space data representingthe virtual space 200A in FIG. 13A based on the virtual room 70A, theavatars 4A to 4C, and the control information on the avatars 4A to 4C(Step S39). Specifically, the control unit 121 generates the virtualspace data representing a virtual space 220A in FIG. 14A. In FIG. 14A,the virtual space 220A includes the avatars 4A, 4B, and 4C and thevirtual room 70A. The virtual room 70A includes the table 5A and thevirtual screen 60A (enlarged first virtual screen).

In FIG. 13A and FIG. 14A, when the virtual room of the avatar 4A isupdated from the virtual room 7A to the virtual room 70A so that thesize of the virtual room of the avatar 4A is enlarged, a virtual screenfor displaying the video content is also updated from the virtual screen6A (of the S size) to the virtual screen 60A (of the M size). That is,the size of the virtual screen arranged in the virtual room is alsoenlarged based on the enlargement of the size of the virtual room. Asthe total number N of avatars increases, the size of the virtual room isfurther enlarged, and the size of the virtual screen is also enlarged,which can encourage a large number of users (avatars) to enjoy viewingthe same video content displayed on the virtual screen within the samevirtual room. Therefore, one user is encouraged to visit the virtualroom of another user (social VR). As the size of the virtual room isenlarged, the size of furniture (for example, table) arranged in thevirtual room may also be enlarged.

In FIG. 13A and FIG. 14A, the avatars 4A and 4B are arranged in thevirtual room 70A so that a relative positional relationship between theavatars 4A and 4B and the virtual room 7A is maintained. Specifically,relative positions of the avatars 4A and 4B with respect to a center ofthe virtual room 70A (or a coordinate space set for the virtual room70A) are substantially the same as relative positions of the avatars 4Aand 4B with respect to a center of the virtual room 7A (or a coordinatespace set for the virtual room 7A). A point of view of the user A(position of the avatar 4A at which the virtual camera is arranged) isprevented from being greatly changed before and after the virtual roomof the avatar 4A is updated from the virtual room 7A to the virtualroom. 70A, which can suitably prevent the user A from suffering from VRsickness. The avatar 4C that is to visit the virtual room of the avatar4A may be arranged at a predetermined position (in this example, apredetermined position between the avatar 4A and the avatar 4B) withinthe virtual room 70A.

Returning to FIG. 12A, the control unit 121 of the user terminal 1Aidentifies the visual field CV of the virtual camera 300, which isillustrated in FIG. 7, based on the position and the inclination of theHMD 110, and then updates the visual-field image displayed on the HMD110 based on the updated virtual space data and the visual field CV ofthe virtual camera 300 (Step S40).

In FIG. 12A, only operation processing of the server 2 and the userterminals 1A and 1C is included for simplicity in description, but oneof ordinary skill in the art would understand that the server 2 alsotransmits the virtual room 70A, the avatar 4C, and the controlinformation on the avatars 4C and 4A to the user terminal 1B. In short,the virtual space provided to the user B is updated as well. The updatedvirtual space includes the table 5A, the virtual room 70A including thevirtual screen 60A, and the avatars 4A to 4C.

With reference to FIG. 12B, a description is given of processingperformed to determine that the size of the virtual room of the avatar4A is not to be enlarged (NO in Step S32). For example, as aprecondition that, when the avatar 4B is not present in the virtual room7A of the user A, the control unit 23 of the server 2 identifies thetotal number N of avatars as two. Then, the control unit 23 determinesthat the size of the virtual room of the avatar 4A is not to be enlargedbased on the total number N of avatars being two and the tableindicating the relationship between the total number N of avatars andthe size of the virtual room, which is shown in Table 1 (NO in StepS32). The control unit 23 transmits the virtual room 7A of the S size(first virtual room), the avatar 4A, and the control information on theavatar 4A to the user terminal 1C (Step S41).

After receiving the virtual room 7A, the avatar 4A, and the controlinformation on the avatar 4A, the control unit 121 of the user terminal1C updates the virtual space data based on the virtual room 7A, theavatars 4A and 4C, and the control information on the avatars 4A and 4C(Step S43). Specifically, the control unit 121 generates the virtualspace data representing the virtual space including the avatars 4A and4C and the virtual room 7A. The control unit 121 updates thevisual-field image displayed on the HMD 110 based on the updated virtualspace data and the visual field CV of the virtual camera 300 (Step S44).

In Step S45, the control unit 23 transmits the avatar 4C and the controlinformation on the avatar 4C to the user terminal 1A. In Step S46, thecontrol unit 121 of the user terminal 1A receives the avatar 4C and thecontrol information on the avatar 4C. The control unit 121 updates thevirtual space data based on the virtual room 7A, the avatars 4A and 4C,and the control information on the avatars 4A and 4C (Step S47).Specifically, the control unit 121 generates the virtual space datarepresenting the virtual space including the avatars 4A and 4C and thevirtual room 7A. The control unit 121 updates the visual-field imagedisplayed on the HMD 110 based on the updated virtual space data and thevisual field CV of the virtual camera 300 (Step S48). A series ofprocessing of at least one embodiment is executed.

According to at least one embodiment, a determination is made whether ornot to update the virtual room of the avatar 4A based on the totalnumber N of avatars currently arranged in the virtual room 7A of theavatar 4A and avatars that are to visit the virtual room 7A, and inresponse to a determination that the virtual room of the avatar 4A is tobe updated, the virtual room of the avatar 4A is updated from thevirtual room 7A (of the S size) to the virtual room 70A (of the M size).The size of the virtual room is enlarged based on the total number N ofavatars, which can prevent a situation unpleasant for the respectiveusers A to C in which a large number of avatars are arranged in thevirtual room 7A of the S size. Therefore, the information processingmethod is capable of improving a virtual experience of the user withinthe social VR.

When the avatar 4C leaves the virtual room 70A under a state under whichthe avatars 4A, 4B, and 4C are arranged in the virtual room 70A, thevirtual room of the avatar 4A may be updated from the virtual room 70Aof the M size to the virtual room 7A of the S size. That is, the size ofthe virtual room of the avatar 4A may be reduced based on the totalnumber N of avatars.

The relationship between the total number N of avatars and the size ofthe virtual room, which is shown in Table 1, is merely an example, andthe size of the virtual room may be gradually enlarged as the totalnumber N of avatars increases one by one (or as avatars enter or exitthe room in groups).

In at least one embodiment, the virtual space data and the visual-fieldimage data are generated on the user terminals 1A to 1C, but may begenerated on the server 2. In this case, the user terminals 1A to 1Ceach display the visual-field image on the HMD 110 based on thevisual-field image data transmitted from the server 2.

With reference to FIG. 10, FIG. 15, and FIG. 16, a description is givenof processing for viewing the video content displayed on the virtualscreen 6A between the users A and B. FIG. 15 is a sequence diagram ofthe processing for viewing the video content according to at least oneembodiment. FIG. 16A is a table of the video content charge tableaccording to at least one embodiment. FIG. 16B is a table of the billingtable according to at least one embodiment. In this description, in FIG.10, the avatar 4A and the avatar 4B are arranged in the virtual room 7Aof the avatar 4A. That is, the user A and the user B share one virtualroom 7A via the communication network 3.

In FIG. 15, in Step S50, the user A executes predetermined operationinput for viewing a video content item V1. For example, the user Aselects an item “View the video content item V1” or other such itemwithin the menu screen displayed on the HMD 110. The control unit 121 ofthe user terminal 1A generates a viewing request signal for requestingthe viewing of the video content item V1 based on the operation inputreceived from the user A, and then transmits the viewing request signalto the server 2.

The control unit 23 of the server 2 receives the viewing request signalfrom the user terminal 1A (Step S51). The control unit 23 acquires thevideo content item V1 from the storage unit (or a server arranged on thecommunication network 3 and configured to store the video content), andthen transmits the acquired video content item V1 to the user terminals1A and 1B (Step S52). The server 2 may transmit the time-divided videocontent item V1. In other words, the server 2 may distribute the videocontent item V1 through streaming.

The control unit 121 of the user terminal 1A receives the video contentitem V1 from the server 2 (Step S53). The control unit 121 updates thevirtual space data based on the video content item V1, the virtual room7A, and the avatars 4A and 4B (Step S54). Specifically, the control unit121 displays the video content item V1 on the virtual screen 6A throughtexture mapping. The control unit 121 generates the virtual space datarepresenting the virtual space 200A including the virtual screen 6A onwhich the video content item V1 is displayed. The control unit 121updates the visual-field image displayed on the HMD 110 based on theupdated virtual space data and the visual field CV of the virtual camera300, in FIG. 7 (Step S55).

The control unit 121 of the user terminal 1B receives the video contentitem V1 from the server 2 (Step S56). The control unit 121 updates thevirtual space data based on the video content item V1, the virtual room7A, and the avatars 4A and 4B (Step S57). Specifically, the control unit121 generates the virtual space data representing the virtual space 200Bincluding the virtual screen 6A on which the video content item V1 isdisplayed. The control unit 121 updates the visual-field image displayedon the HMD 110 based on the updated virtual space data and the visualfield CV of the virtual camera 300, which is illustrated in FIG. 7 (StepS58).

The control unit 23 of the server 2 updates the billing table stored inthe storage unit 22, which is shown FIG. 16B (Step S59). Specifically,the control unit 23 refers to the video content charge table stored inthe storage unit 22, in FIG. 16A, to identify a viewing unit price forthe video content item V1 (in this example, the viewing unit price forthe video content item V1 is 300 yen). The control unit 23 updatesbilling data on the users A and B within the billing table based on theviewing unit price for the video content item V1. In FIG. 16, theviewing unit price may differ depending on the kind of video content (inthis example, the viewing unit price for a video content item V2 is 500yen). In addition, when the video content is viewed by a plurality ofavatars via one virtual room, a group discount may be applied for theviewing unit price for the video content. For example, when the videocontent item V1 is viewed by avatars whose number is equal to or largerthan a predetermined number via one virtual room (for example, thevirtual room 7A), the viewing unit price for the video content item V1may be reduced from 300 yen to 200 yen. In the same manner, when thevideo content item V2 is viewed by the avatars whose number is equal toor larger than the predetermined number via one virtual room (forexample, the virtual room 7A), the viewing unit price for the videocontent item V2 may be reduced from 500 yen to 300 yen.

With reference to FIG. 17 and FIG. 18, a description is given ofprocessing for enlarging the size of the virtual screen 6A (firstvirtual screen) without updating the size of the virtual room. FIG. 17is a sequence diagram of the processing for enlarging the virtual screen6A (specifically, processing for updating the virtual screen 6A of the Ssize to the virtual screen 60A of the M size (enlarged first virtualscreen) according to at least one embodiment. FIG. 18A is a diagram ofthe virtual space 200A provided to the user A before the virtual screen6A is enlarged according to at least one embodiment. FIG. 18B is adiagram of the virtual space 200A provided to the user A after thevirtual screen 6A is enlarged according to at least one embodiment. Inthis description, in FIG. 18A, the avatar 4A and the avatar 4B arearranged in the virtual room 7A of the avatar 4A. That is, the user Aand the user B share one virtual room 7A via the communication network3.

In FIG. 17, in Step S60, the user A executes predetermined operationinput for enlarging the virtual screen 6A. For example, the user Aselects an item “enlarge the size of the virtual screen to the M size”or other such item within the menu screen displayed on the HMD 110. Thecontrol unit 121 of the user terminal 1A transmits a screen enlargementrequest signal for enlarging a size of the virtual screen 6A to the Msize to the server 2 based on the operation input received from the userA.

The control unit 23 of the server 2 receives the screen enlargementrequest signal from the user terminal 1A (Step S61). The control unit 23acquires the virtual screen 60A of the M size from the storage unit 22,and then transmits the virtual screen 60A (enlarged virtual screen) tothe user terminals 1A and 1B (Step S62).

The control unit 121 of the user terminal 1A receives the virtual screen60A from the server 2 (Step S63). The control unit 121 updates thevirtual space data based on the virtual screen 60A, the virtual room 7A,and the avatars 4A and 4B (Step S64). Specifically, the control unit 121updates the virtual screen from the virtual screen 6A of the S size tothe virtual screen 60A of the M size, and then generates the virtualspace data representing the virtual space 200A including: the virtualroom 7A including the virtual screen 60A; and the avatars 4A and 4B. Thecontrol unit 121 updates the visual-field image displayed on the HMD 110based on the updated virtual space data and the visual field CV of thevirtual camera 300, in FIG. 7 (Step S65).

In Step S66, the control unit 121 of the user terminal 1B receives thevirtual screen 60A from the server 2. The control unit 121 updates thevirtual space data based on the virtual screen 60A, the virtual room 7A,and the avatars 4A and 4B (Step S67). Specifically, the control unit 121updates the virtual screen from the virtual screen 6A of the S size tothe virtual screen 60A of the M size, and then generates the virtualspace data representing the virtual space. The virtual space includesthe virtual room 7A including the virtual screen 60A; and the avatars 4Aand 4B. The control unit 121 updates the visual-field image displayed onthe HMD 110 based on the updated virtual space data and the visual fieldCV of the virtual camera 300, in FIG. 7 (Step S68).

In Step S69, the server 2 updates the billing table stored in thestorage unit 22, which is shown in FIG. 16B. Specifically, the controlunit 23 refers to the video content charge table, which is shown in FIG.16A, to identify an additional charge (virtual screen enlargementcharge) required for enlarging the size of the virtual screen to the Msize based on a screen enlargement request. For example, when the videocontent being currently viewed is the video content item V1, the controlunit 23 identifies the virtual screen enlargement charge required forenlarging the size of the virtual screen to the M size as 500 yen. Thecontrol unit 23 updates the billing data on the user A within thebilling table based on the identified virtual screen enlargement charge(for example, 500 yen). In FIG. 16B, when the user A enlarges the sizeof the virtual screen to the M size through a predetermined operationwhile the users A and B are viewing the video content item V1 on thevirtual screen 6A of the S size, an amount to be billed to the user Afor the viewing of the video content item V1 is (300 yen (viewing unitprice for the video content item V1))+(500 yen (virtual screenenlargement charge))=(800 yen). The virtual screen enlargement chargemay be borne only by the user A who has requested for the enlargement ofthe virtual screen, or may be equally shared by the users A and B.

According to at least one embodiment, the virtual screen is updated sothat the size of the virtual screen of the avatar 4A is enlarged to theM size based on the screen enlargement request signal, and a billingamount to be billed to the user A is updated. Even when there is noavatar that is to visit the virtual room 7A of the avatar 4A, the user Acan enjoy viewing video content or other such moving image on a largevirtual screen by paying a predetermined amount.

According to at least one embodiment, the billing amount billed to theuser A when the size of the virtual screen is enlarged to the M size(from the virtual screen 6A to the virtual screen 60A) based on theenlargement of the size of the virtual room of the avatar 4A (from thevirtual room 7A to the virtual room 70A) be smaller than the billingamount billed to the user A when the size of the virtual screen isenlarged to the M size based on the screen enlargement request signal.For example, the billing amount based on the enlargement of the virtualroom is, for example, 0 yen. The billing amount based on the screenenlargement request signal is 500 yen as in FIG. 16A. The billing amountbased on the enlargement of the virtual room is thus smaller than thebilling amount based on the screen enlargement request signal, which canencourage a large number of users to enjoy viewing the same videocontent within the same virtual room. Therefore, one user is encouragedto visit the virtual room of another user (social VR).

With reference to FIG. 19 and FIG. 20, a description is given of anexample of the processing for synchronizing the movements of therespective avatars 4A and 4B between the user terminal 1A and the userterminal 1B. FIG. 19A is a diagram of the virtual space 200A provided tothe user A according to at least one embodiment. FIG. 19B is a diagramof the virtual space 200B provided to the user B according to at leastone embodiment. FIG. 20 is a sequence diagram of the processing forsynchronizing the movements of the respective avatars 4A and 4B betweenthe user terminal 1A and the user terminal 1B according to at least oneembodiment. In this description, in FIGS. 19A-B, the avatar 4Aassociated with the user terminal 1A (user A) and the avatar 4Bassociated with the user terminal 1B (user B) share the same virtualspace. That is, the user A and the user B share one virtual space viathe communication network 3.

In FIG. 19A, the virtual space 200A of the user A includes the avatar4A, the avatar 4B, and the virtual screen 6A. The avatar 4A is operatedby the user A, and is configured to move in association with the actionof the user A. The avatar 4B is operated by the user B, and isconfigured to move in association with the action of the user B.

For example, the positions of the avatars 4A and 4B may be identifiedbased on the positions of the HMDs 110 of the user terminals 1A and 1B.In the same manner, the directions of the faces of the avatars 4A and 4Bmay be identified based on the inclinations of the HMDs 110 of the userterminals 1A and 1B. In addition, the actions of the hands of theavatars 4A and 4B may be identified based on the actions of the externalcontrollers of the user terminals 1A and 1B. Relative positions of theirises and pupils with respect to the whites of the eyes of the avatars4A and 4B may be identified based on the lines of sight of the users Aand B, which are detected by the eye gaze sensor 140. In particular, therelative position of the iris and pupil of the left eye with respect tothe white of the left eye of the avatar may be identified based on datarepresenting the line of sight of the left eye of the user, which isdetected by the eye gaze sensor for a left eye. In addition, therelative position of the iris and pupil of the right eye with respect tothe white of the right eye of the avatar may be identified based on datarepresenting the line of sight of the right eye of the user, which isdetected by the eye gaze sensor for a right eye.

The lines of sight of the avatars 4A and 4B may be identified based onthe inclinations of the HMDs 110 of the user terminals 1A and 1B. Inother words, the line of sight of the avatar may be identified based onthe visual axis L of the virtual camera. In this case, the line of sightof the avatar represents the lines of sight of both the eyes of theavatar. The lines of sight of the avatars 4A and 4B may be identifiedbased on the inclinations of the HMDs 110 of the user terminals 1A and1B and the lines of sight of the users A and B (lines of sight of boththe eyes of the users A and B). In this case, the lines of sight of theusers A and B is identified based on data representing the lines ofsight of the users A and B detected by the eye gaze sensor 140 asdescribed above.

The virtual camera 300 in FIG. 6 may be arranged at each of the eyes ofthe avatars 4A and 4B. In particular, the left-eye virtual camera may bearranged at each of the left eyes of the avatars 4A and 4B, while theright-eye virtual camera may be arranged at each of the right eyes ofthe avatars 4A and 4B. In the following description, it is assumed thatthe virtual camera 300 is arranged at each of the eyes of the avatars 4Aand 4B. Therefore, the visual field CV of the virtual camera 300, inFIG. 7, matches the visual field CV of each of the avatars 4A and 4B.

The virtual screen 6A is configured to display the video content (movingimage). The control unit 121 is capable of displaying the video contenton a surface of the virtual screen 6A through the texture mapping. Eachof the users A and B can enjoy the video content displayed on thevirtual screen 6A through the visual-field image displayed on the HMD110.

In FIG. 19B, the virtual space 200B of the user B includes the avatar4A, the avatar 4B, and the virtual screen 6A. The positions of theavatars 4A and 4B within the virtual space 200A may correspond to thepositions of the avatars 4A and 4B within the virtual space 200B.

With reference to FIG. 20, in Step S70, the control unit 121 of the userterminal 1A generates voice data on the user A. For example, when theuser A inputs voice to the microphone 118 of the user terminal 1A, themicrophone 118 generates voice data representing the input voice. Themicrophone 118 transmits the generated voice data to the control unit121 via the I/O interface 124.

In Step S71, the control unit 121 of the user terminal 1A generates thecontrol information on the avatar 4A, and then transmits the generatedcontrol information on the avatar 4A and the voice data representing thevoice of the user A (voice data on the user A) to the server 2. Thecontrol unit 23 of the server 2 receives the control information on theavatar 4A and the voice data on the user A from the user terminal 1A(Step S72). In this case, the control information on the avatar 4Arepresents information required for controlling the action of the avatar4A. The control information on the avatar 4A includes the informationrelating to the line of sight (line-of-sight information) of the avatar4A. In addition, the control information on the avatar 4A may furtherinclude the information relating to the position of the avatar 4A, theinformation relating to the direction of the face of the avatar 4A, theinformation relating to the action of the hand of the avatar 4A, andinformation relating to the movement of the eye (in particular, themovement of the iris and pupil) of the avatar 4A.

In Step S73, the control unit 121 of the user terminal 1B generates thecontrol information on the avatar 4B, and then transmits the generatedcontrol information on the avatar 4B to the server 2. The control unit23 of the server 2 receives the control information on the avatar 4Bfrom the user terminal 1B (Step S74). In this case, the controlinformation on the avatar 4B represents information required forcontrolling the action of the avatar 4B. The control information on theavatar 4B includes the information relating to the line of sight(line-of-sight information) of the avatar 4B. In addition, the controlinformation on the avatar 4B may further include the informationrelating to the position of the avatar 4B, the information relating tothe direction of the face of the avatar 4B, the information relating tothe action of the hand of the avatar 4B, and information relating to themovement of the eye (in particular, the movement of the iris and pupil)of the avatar 4B.

The server 2 transmits the control information on the avatar 4B to theuser terminal 1A (Step S75), while transmitting the control informationon the avatar 4A and the voice data on the user A to the user terminal1B (Step S79). After receiving the control information on the avatar 4Bin Step S76, the control unit 121 of the user terminal 1A updates thevirtual space data representing the virtual space 200A in FIG. 19A basedon the control information on the avatars 4A and 4B (Step S77). Forexample, when the avatars 4A and 4B move, the virtual space datarepresenting the virtual space including the avatars 4A and 4B that havemoved is generated. The control unit 121 of the user terminal 1Aidentifies the visual field CV of the avatar 4A (virtual camera 300)based on the position and the inclination of the HMD 110, and thenupdates the visual-field image displayed on the HMD 110 based on theupdated virtual space data and the visual field CV of the avatar 4A(Step S78).

After receiving the control information on the avatar 4A and the voicedata on the user A in Step S80, the control unit 121 of the userterminal 1B updates the virtual space data representing the virtualspace 200B in FIG. 19B based on the control information on the avatars4A and 4B (Step S81). The control unit 121 of the user terminal 1Bidentifies the visual field CV of the avatar 4B (virtual camera 300)based on the position and the inclination of the HMD 110, and thenupdates the visual-field image displayed on the HMD 110 based on theupdated virtual space data and the visual field CV of the avatar 4B(Step S82).

The control unit 121 of the user terminal 1B processes the voice data onthe user A based on the received voice data on the user A, theinformation relating to the position of the avatar 4A, which is includedin the control information on the avatar 4A, and a predetermined voiceprocessing algorithm. The control unit 121 transmits the processed voicedata to the headphones 116, and then the headphones 116 output the voiceof the user A based on the processed voice data (Step S83). A voice chatbetween the users (avatars) in the virtual space is possible.

In at least one embodiment, after the user terminals 1A and 1B transmitthe control information on the avatar 4A and the control information onthe avatar 4B, respectively, to the server 2, the server 2 transmits thecontrol information on the avatar 4A to the user terminal 1B, andtransmits the control information on the avatar 4B to the user terminal1A. The movements of the respective avatars 4A and 4B can besynchronized between the user terminal 1A and the user terminal 1B. Therespective processing steps in FIG. 20 may be executed for every frame.

In the case of synchronizing the movements of the respective avatars 4Ato 4C are to be synchronized among the user terminals 1A to 1C, the userterminal 1A transmits the control information on the avatar 4A to theserver 2, the user terminal 1B transmits the control information on theavatar 4B to the server 2, and the user terminal 1C transmits thecontrol information on the avatar 4C to the server 2. The server 2transmits the control information on the avatars 4A and 4C to the userterminal 1B, transmits the control information on the avatars 4B and 4Cto the user terminal 1A, and transmits the control information on theavatars 4A and 4B to the user terminal 1C.

With reference to FIG. 21 to FIG. 24, a description is given of aninformation processing method according to at least one embodiment ofthis disclosure. FIG. 21 is a flowchart of the information processingmethod according to at least one embodiment. FIG. 22 is a plan view ofthe virtual space 200A provided to the user A before voice is input tothe microphone 118 of the user terminal 1B according to at least oneembodiment. FIG. 23 is a diagram of the visual-field image V displayedon the HMD 110 of the user terminal 1A before the voice is input to themicrophone 118 of the user terminal 1B according to at least oneembodiment. FIG. 24 is a diagram of the visual-field image V displayedon the HMD 110 of the user terminal 1A after the voice is input to themicrophone 118 of the user terminal 1B according to at least oneembodiment.

In at least one embodiment, in FIG. 22, the avatar 4A, the avatar 4B,and the avatar 4C associated with the user terminal 1C (user C) sharethe same virtual space. That is, the users A to C share one virtualspace via the communication network 3. The avatar 4C is operated by theuser C, and is configured to move in association with the action of theuser C. The avatars 4A to 4C are viewing the video content displayed onthe display screen 60A of the virtual screen 6A. In FIG. 22, a line ofsight SLa of the avatar 4A, which is not visualized, indicates apredetermined spot on the display screen 60A.

The virtual space 200A of the user A includes the avatar 4A, the avatar4B, the avatar 4C, and the virtual screen 6A. The control unit 121 ofthe user terminal 1A generates the virtual space data representing thevirtual space 200A.

With reference to FIG. 21, in Step S90, the control unit 121 of the userterminal 1A (hereinafter referred to simply as “control unit 121”)identifies the visual field CV of the avatar 4A. As described above, thevisual field CV of the avatar 4A matches the visual field CV of thevirtual camera arranged at the eye of the avatar 4A. The control unit121 updates the visual field CV of the avatar 4A in accordance with themovement of the HMD 110 of the user terminal 1A. Processing for updatingthe visual field CV of the avatar 4A may be executed for every frame. Inaddition, the control unit 121 updates the visual-field image Vdisplayed on the HMD 110 based on the updated visual field CV and thevirtual space data representing the virtual space 200A. Processing forupdating the visual-field image may be executed for every frame as well.

In Step S91, the control unit 121 receives the control information onthe avatar 4B, which is required for controlling the action of theavatar 4B, from the server 2. In this case, the control information onthe avatar 4B includes the information relating to the line of sight(line-of-sight information) of the avatar 4B. Specifically, afterreceiving the control information on the avatar 4B from the userterminal 1B, the server 2 transmits the control information on theavatar 4B to the user terminals 1A and 1C. The control unit 121 receivesthe control information on the avatar 4B from the server 2.

In Step S92, the control unit 121 receives the control information onthe avatar 4C, which is required for controlling the action of theavatar 4C, from the server 2. In this case, the control information onthe avatar 4C includes the line-of-sight information on the avatar 4C.Specifically, after receiving the control information on the avatar 4Cfrom the user terminal 1C, the server 2 transmits the controlinformation on the avatar 4C to the user terminals 1A and 1B. Thecontrol unit 121 receives the control information on the avatar 4C fromthe server 2.

It is to be noted that, before the processing of Step S91, the controlunit 121 transmits the control information on the avatar 4A, which isrequired for controlling the action of the avatar 4A, to the server 2.

The control unit 121 determines whether or not the avatar 4B existswithin the visual field CV of the avatar 4A based on the informationrelating to the position of the avatar 4B, which is included in thecontrol information on the avatar 4B (Step S93). When determining thatthe avatar 4B does not exist within the visual field CV of the avatar 4A(NO in Step S93), the control unit 121 generates a line-of-sight object5B of the avatar 4B, in FIG. 22, based on the line-of-sight informationon the avatar 4B (Step S94).

For example, in FIG. 22, the control unit 121 may generate theline-of-sight object 5B based on the line of sight of the avatar 4B andan intersecting point Ca, which is an intersection between the line ofsight of the avatar 4B and the display screen 60A of the virtual screen6A. In particular, the control unit 121 may identify the direction ofthe line-of-sight object 5B based on the direction of the line of sightof the avatar 4B, and may identify the length of the line-of-sightobject 5B based on a distance between the position of the avatar 4B andthe intersecting point Ca.

When the control unit 121 determines that the avatar 4B exists withinthe visual field CV of the avatar 4A (YES in Step S93), the procedureadvances to the processing of Step S95. In Step S95, the control unit121 determines whether or not the avatar 4C exists within the visualfield CV of the avatar 4A based on the information relating to theposition of the avatar 4C, which is included in the control informationon the avatar 4C (Step S95). When determining that the avatar 4C doesnot exist within the visual field CV of the avatar 4A (NO in Step S95),the control unit 121 generates a line-of-sight object 5C of the avatar4C, which is illustrated in FIG. 22, based on the line-of-sightinformation on the avatar 4C (Step S96).

For example, in FIG. 22, the control unit 121 may generate theline-of-sight object 5C based on the line of sight of the avatar 4C andan intersecting point Cb, which is an intersection between the line ofsight of the avatar 4C and the display screen 60A of the virtual screen6A. In particular, the control unit 121 may identify the direction ofthe line-of-sight object 5C based on the direction of the line of sightof the avatar 4C, and may identify the length of the line-of-sightobject 5C based on a distance between the position of the avatar 4C andthe intersecting point Cb. When the control unit 121 determines that theavatar 4C exists within the visual field CV of the avatar 4A (YES inStep S95), the procedure advances to the processing of Step S97.

The line-of-sight objects 5B and 5C are generated, and hence the linesof sight of the avatars 4B and 4C are visualized in the virtual space200A. For example, in FIG. 23, the lines of sight of the avatars 4B and4C (namely, the line-of-sight objects 5B and 5C), which have beenvisualized, are displayed in the visual-field image V displayed on theHMD 110 of the user terminal 1A.

In Step S97, the control unit 121 receives the voice data representingthe voice of the user B (voice data on the user B). For example, whenthe user B inputs voice (for example, “Hang in there!”) to themicrophone 118 of the user terminal 1B, the microphone 118 generatesvoice data representing the input voice, and then transmits thegenerated voice data to the control unit 121 of the user terminal 1B viathe I/O interface 124. After the control unit 121 of the user terminal1B transmits the voice data to the server 2, the server 2 transmits thevoice data to the user terminals 1A and 1C. The control unit 121 of theuser terminal 1A receives the voice data on the user B from the server2. The control unit 121 processes the voice data on the user B based onthe received voice data on the user B, the information relating to theposition of the avatar 4B, which is included in the control informationon the avatar 4B, and a predetermined voice processing algorithm. Thecontrol unit 121 transmits the processed voice data to the headphones116, and then the headphones 116 output the voice of the user B based onthe processed voice data.

The control unit 121 updates the line-of-sight object 5B so as to changea display form of the line-of-sight object 5B (Step S98). In thisregard, as a change in display form of the line-of-sight object 5B, thecontrol unit 121 may continuously change a shape and a color of theline-of-sight object 5B on a time axis. For example, in FIG. 24, thecontrol unit 121 may change the shape of the line-of-sight object 5B ina wave-like form. In particular, the wave-like form of the line-of-sightobject 5B may be continuously changed on the time axis.

When the updated line-of-sight object 5B and the line-of-sight object 5Cexist within the visual field of the avatar 4A, as illustrated in FIG.24, the control unit 121 displays the visual-field image V including theupdated line-of-sight object 5B and the line-of-sight object 5C on theHMD 110 of the user terminal 1A.

According to at least one embodiment, the voice of the user B (“Hang inthere!”) is input when the microphone 118 of the user terminal 1B, thevoice data on the user B is received from the user terminal 1B via theserver 2. The line-of-sight object 5B is updated so that the displayform of the line-of-sight object 5B of the avatar 4B is changed, and thevisual-field image V including the updated line-of-sight object 5B andthe line-of-sight object 5C is displayed on the HMD 110 of the userterminal 1A. Therefore, the user A can easily grasp the fact that thevoice has been output from the user B associated with the avatar 4B byvisually recognizing a change in display form of the line-of-sightobject 5B. That is, the user A can easily grasp which user is currentlytalking based on the change in display form of the line-of-sight object5B. Therefore, the information processing method is capable of improvinga virtual experience of the user within the virtual space shared by aplurality of users.

According to at least one embodiment, when the avatar 4B does not existwithin the visual field CV of the avatar 4A, the line-of-sight object 5Bis generated based on the line of sight of the avatar 4B and theintersecting point Ca. In addition, when the avatar 4C does not existwithin the visual field CV of the avatar 4A, the line-of-sight object 5Cis generated based on the line of sight of the avatar 4C and theintersecting point Cb. Even when the avatars 4B and 4C are not displayedwithin the visual-field image V, the user A can easily grasp the factthat the voice has been output from the user B relating to the avatar 4Bby visually recognizing the change in display form of the line-of-sightobject 5B.

In at least one embodiment, when the avatar 4B (4C) exists within thevisual field CV of the avatar 4A, the user A can relatively easilyidentify which user is currently talking, and hence the line-of-sightobject 5B (5C) is not generated. However, even when the avatar 4B (4C)exists within the visual field CV of the avatar 4A, the control unit 121may generate the line-of-sight object 5B (5C).

With reference to FIG. 25 to FIG. 27, a description is given of aninformation processing method according to at least one embodiment ofthis disclosure. FIG. 25 is a flowchart of the information processingmethod according to at least one embodiment. FIG. 26 is a plan view ofthe virtual space 200A provided to the user A after voice is input tothe microphone 118 of the user terminal 1B according to at least oneembodiment. FIG. 27 is a diagram of the visual-field image V displayedon the HMD 110 of the user terminal 1A after the voice is input to themicrophone 118 of the user terminal 1B according to at least oneembodiment. The situation with respect to FIG. 25 to FIG. 27 is similarto the situation with respect to FIG. 21 to FIG. 24.

With reference to FIG. 25, in Step S100, the control unit 121 of theuser terminal 1A (hereinafter referred to simply as “control unit 121”)identifies the visual field CV of the avatar 4A. The control unit 121receives the control information on the avatar 4B, which includes theline-of-sight information on the avatar 4B, from the server 2 (StepS101). The control unit 121 receives the control information on theavatar 4C, which includes the line-of-sight information on the avatar4C, from the server 2 (Step S102). Before the processing of Step S101,the control unit 121 transmits the control information on the avatar 4Ato the server 2.

In Step S103, after receiving the voice data representing the voice ofthe user B (for example, “Hang in there!”) (voice data on the user B)from the server 2, the control unit 121 processes the voice data on theuser B. The headphones 116 output the voice of the user B based on theprocessed voice data.

The control unit 121 determines that the user who has output the voiceis the user B, and then determines whether or not the avatar 4B existswithin the visual field CV of the avatar 4A based on the informationrelating to the position of the avatar 4B, which is included in thecontrol information on the avatar 4B (Step S104). When determining thatthe avatar 4B does not exist within the visual field CV of the avatar 4A(NO in Step S104), the control unit 121 generates the line-of-sightobject 5B of the avatar 4B, in FIG. 26, based on the line-of-sightinformation on the avatar 4B (Step S105). The method of generating theline-of-sight object 5B is as described above. When the determination ofStep S104 results in YES, the line-of-sight object 5B of the avatar 4Bis not generated.

Only the line-of-sight object 5B of the avatar 4B associated with theuser B who has output the voice is generated, and hence only the line ofsight of the avatar 4B is visualized in the virtual space 200A. Forexample, in FIG. 27, only the visualized line of sight of the avatar 4B(namely, the line-of-sight object 5B) is displayed in the visual-fieldimage V displayed on the HMD 110 of the user terminal 1A. Aline-of-sight object corresponding to a line of sight SLc of the avatar4C, in FIG. 26, is not generated, and hence the line of sight SLc of theavatar 4C is not visualized.

According to at least one embodiment, when the line-of-sight object 5Bexists within the visual field CV of the avatar 4A, in FIG. 27, thecontrol unit 121 displays the visual-field image V including theline-of-sight object 5B on the HMD 110 of the user terminal 1A.Therefore, the user A can easily grasp the fact that the voice has beenoutput from the user B associated with the avatar 4B by visuallyrecognizing the line-of-sight object 5B displayed within thevisual-field image V.

According to at least one embodiment, when the avatar 4B does not existwithin the visual field CV of the avatar 4A, the line-of-sight object 5Bis generated. Even when the avatar 4B is not displayed within thevisual-field image V, the user A can easily grasp the fact that thevoice has been output from the user B associated with the avatar 4B byvisually recognizing the line-of-sight object 5B. Even when the avatar4B exists within the visual field CV of the avatar 4A, the control unit121 may generate the line-of-sight object 5B.

With reference to FIG. 28 to FIG. 30, a description is given of aninformation processing method according to at least one embodiment ofthis disclosure. FIG. 28 is a flowchart of the information processingmethod according to at least one embodiment. FIG. 29A is a plan view ofthe virtual space 200A provided to the user A, which indicates a stateunder which the avatar 4B is talking to the avatar 4A according to atleast one embodiment. FIG. 29B is a plan view of the virtual space 200Aprovided to the user A, which indicates a state under which the avatar4A has turned to face the avatar 4B according to at least oneembodiment. FIG. 30A is a diagram of a state of the face of the avatar4B in FIG. 29A according to at least one embodiment. FIG. 30B is adiagram of a state of the face of the avatar 4B in FIG. 29B according toat least one embodiment. The situation in FIG. 28 to FIG. 30B is similarto the situation in FIG. 21 to FIG. 24 except for the position ofarrangement of each avatar.

With reference to FIG. 28, in Step S110, the control unit 121 of theuser terminal 1A (hereinafter referred to simply as “control unit 121”)identifies the visual field CV of the avatar 4A. The control unit 121receives the control information on the avatar 4B, which includes theline-of-sight information on the avatar 4B, from the server 2 (StepS111). The control unit 121 receives the control information on theavatar 4C, which includes the line-of-sight information on the avatar4C, from the server 2 (Step S112). Before the processing of Step S111,the control unit 121 transmits the control information on the avatar 4Ato the server 2.

In Step S113, after receiving the voice data representing the voice ofthe user B (for example, “Hey”) (voice data on the user B) from theserver 2, the control unit 121 processes the voice data on the user B.The headphones 116 output the voice of the user B based on the processedvoice data.

The control unit 121 determines that the user who has output the voiceis the user B, and then determines whether or not the voice data on theuser B has been received with a line of sight SLb of the avatar 4B beingdirected toward the avatar 4A (Step S114). Specifically, the controlunit 121 determines whether or not the line of sight SLb of the avatar4B intersects with the avatar 4A based on the line-of-sight informationon the avatar 4B and information relating to the positions of theavatars 4B and 4A during a period during which the voice data on theuser B is being received from the server 2. When the determination ofStep S114 results in YES, the control unit 121 determines whether or notthe avatar 4B exists within the visual field CV of the avatar 4A (StepS115). When the determination of Step S115 results in NO, the controlunit 121 updates the avatar 4B so as to change the display form of theavatar 4B (Step S116).

For example, in FIG. 30B, the control unit 121 may change a color of theface of the avatar 4B. The control unit 121 may change a color, a shape,or the like of a predetermined body part of the avatar 4B other than theface. When the avatar 4A turns around toward the avatar 4B in responseto calling (“Hey”) from the avatar 4B, in FIG. 30B, the updated avatar4B (avatar 4B with the face having been changed in color) and the avatar4C are positioned within the visual field CV of the avatar 4A. That is,the avatar 4C and the visual-field image including the avatar 4B withthe face having been changed in color are displayed on the HMD 110 ofthe user terminal 1A, and hence the user A can easily grasp the factthat the avatar who has talked to the avatar 4A is the avatar 4B byvisually recognizing the avatar 4B with the face having been changed incolor. According to at least embodiment, the user A can easily grasp thefact that the voice has been output from the user B associated with theavatar 4B by visually recognizing the change in display form of theavatar 4B. When the determination of Step S114 results in NO or when thedetermination of Step S115 results in YES, the display form of theavatar 4B is not changed.

According to at least one embodiment, when the avatar 4B does not existwithin the visual field CV of the avatar 4A, the display form of theavatar 4B is changed. Even in a case where the avatar 4B is notdisplayed within the visual-field image V when the user A is talked toby the avatar 4B, the user A can easily grasp the fact that the voicehas been output from the user B associated with the avatar 4B later byvisually recognizing the change in display form of the avatar 4B. Evenwhen the avatar 4B exists within the visual field CV of the avatar 4A,the control unit 121 may change the display form of the avatar 4B.

The above descriptions of embodiments presuppose that the virtual spacedata representing the virtual space 200A is updated by the user terminal1A, but the virtual space data may be updated by the server 2. Further,the above descriptions of embodiments presuppose that the visual-fieldimage data corresponding to the visual-field image V is updated by theuser terminal 1A, but the visual-field image data may be updated by theserver 2. In this case, the user terminal 1A displays the visual-fieldimage on the HMD 110 based on the visual-field image data transmittedfrom the server 2.

In order to implement various types of processing to be executed by thecontrol unit 23 of the server 2 (or the control unit 121 of the userterminal 1) with use of software, a control program for executingvarious kinds of processing on a computer (processor) may be installedin advance into the storage unit 22 (or the storage unit 123) or thememory. Alternatively, the control program may be stored in acomputer-readable storage medium, for example, a magnetic disk (HDD orfloppy disk), an optical disc (for example, CD-ROM, DVD-ROM, or Blu-raydisc), a magneto-optical disk (for example, MO), and a flash memory (forexample, SD card, USB memory, or SSD). In this case, the storage mediumis connected to the server 2 (or the control device 120), and hence theprogram stored in the storage medium is installed into the storage unit22 (or the storage unit 123). Then, the control program installed in thestorage unit 22 (or the storage unit 123) is loaded onto the RAM, andthe processor executes the loaded program. In this manner, the controlunit 23 (or the control unit 121) executes the various kinds ofprocessing.

The control program may be downloaded from a computer on thecommunication network 3 via the communication interface. Also in thiscase, the downloaded program is similarly installed into the storageunit 22 (or the storage unit 123).

This concludes description of some embodiments of this disclosure.However, the description is not to be read as a restrictiveinterpretation of the technical scope of this disclosure. The abovedescribed embodiments are merely given as an example, and a personskilled in the art would understand that various modifications can bemade to the embodiment within the scope of this disclosure set forth inthe appended claims. Thus, the technical scope of this disclosure is tobe defined based on the scope of this disclosure set forth in theappended claims and an equivalent scope thereof.

[Supplementary Note 1]

(1) An information processing method to be executed by a computer in avirtual space distribution system. The virtual space distribution systemincludes a plurality of user terminals, each including a head-mounteddevice to be worn on a head of a user; and a server. The informationprocessing method includes generating first virtual space data fordefining a first virtual space. The first virtual space includes a firstavatar associated with a first user terminal among the plurality of userterminals; and a first virtual room in which the first avatar isarranged. The method further includes receiving, from a second userterminal among the plurality of user terminals, a visiting requestsignal for requesting that a second avatar associated with the seconduser terminal visit the first virtual room. The method further includesdetermining, in response to receiving the visiting request, whether ornot the first virtual room is to be updated based on a total number ofavatars currently arranged in the first virtual room and second avatars.The method further includes updating, in response to a determinationthat the first virtual room is to be updated, the first virtual room soas to update a size of the first virtual room.

(2) An information processing method according to Item (1), furtherincluding arranging the first avatar in the updated first virtual roomso as to maintain a relative positional relationship between the firstavatar and the first virtual room.

(3) An information processing method according to Item (1) or (2), inwhich the first virtual room includes a first virtual screen configuredto display a moving image. The updating includes updating the firstvirtual room so as to enlarge a size of the first virtual screen basedon enlargement of the size of the first virtual room.

(4) An information processing method according to any one of Items (1)to (3), in which the first virtual room includes a first virtual screenconfigured to display a moving image. The information processing methodfurther includes receiving, from the first user terminal, a screenenlargement request signal for requesting that a size of the firstvirtual screen be enlarged to a predetermined size. The method furtherincludes updating the first virtual screen so as to enlarge the size ofthe first virtual screen to the predetermined size based on the screenenlargement request signal.

(5) An information processing method according to Item (4), furtherincluding determining a billing amount to be billed to a first userwearing the head-mounted device of the first user terminal for viewingof the moving image performed by the first user. Determining the billingamount includes updating the billing amount to be billed to the firstuser based on the screen enlargement request signal.

(6) An information processing method according to Item (5), in which theupdating includes updating the first virtual room so as to enlarge thesize of the first virtual screen based on enlargement of the size of thefirst virtual room. The first user is billed a first billing amount whenthe size of the first virtual screen is enlarged to a predetermined sizebased on the enlargement of the size of the first virtual room, thefirst billing amount being smaller than a second billing amount to bebilled to the first user when the size of the first virtual screen isenlarged to the predetermined size based on the screen enlargementrequest signal.

(7) A system for executing the information processing method of any oneof Items (1) to (6).

[Supplementary Note 2]

A description is given of outlines of embodiments of this disclosure.

(11) An information processing method to be executed by a computer in avirtual space distribution system. The virtual space distribution systemincludes a plurality of user terminals, each including a head-mounteddevice, a voice input unit for inputting voice, and a voice output unitfor outputting voice; and a server. The information processing methodincludes generating virtual space data for defining a virtual space. Thevirtual space includes a first avatar associated with a first userterminal among the plurality of user terminals; a second avatarassociated with a second user terminal among the plurality of userterminals; and a third avatar associated with a third user terminalamong the plurality of user terminals. The method further includesupdating a visual field of the first avatar in accordance with amovement of the head-mounted device of the first user terminal. Themethod further includes displaying a visual-field image on thehead-mounted device of the first user terminal based on the visual fieldof the first avatar and the virtual space data. The method furtherincludes receiving, when voice is input to the voice input unit of thesecond user terminal, voice data representing the input voice from thesecond user terminal. The method further includes changing, in responseto receiving the input voice, one of a display form of the second avatarand a display form of a line of sight of the second avatar in thevisual-field image displayed on the head-mounted device of the firstuser terminal.

(12) An information processing method according to Item (11), furtherincluding generating a second line-of-sight object based on the line ofsight of the second avatar. The method further includes generating athird line-of-sight object based on a line of sight of the third avatar.The changing includes updating, in response to receiving the inputvoice, the second line-of-sight object so as to change the display formof the second line-of-sight object. The changing further includesdisplaying, when the updated second line-of-sight object and the thirdline-of-sight object exist within the visual field of the first avatar,the visual-field image including the updated second line-of-sight objectand the third line-of-sight object on the head-mounted device of thefirst user terminal.

(13) An information processing method according to Item (12), furtherincluding executing the generating of the second line-of-sight objectwhen the second avatar does not exist within the visual field of thefirst avatar. The method further includes executing generating of thethird line-of-sight object when the third avatar does not exist withinthe visual field of the first avatar.

(14) An information processing method according to Item (11), in whichthe changing includes generating, in response to receiving the inputvoice, a second line-of-sight object based on the line of sight of thesecond avatar. The chaining further includes displaying, when the secondline-of-sight object exists within the visual field of the first avatar,the visual-field image including the second line-of-sight object on thehead-mounted device of the first user terminal.

(15) An information processing method according to Item (14), furtherincluding executing the generating of the second line-of-sight objectwhen the second avatar does not exist within the visual field of thefirst avatar.

(16) An information processing method according to Item (11), in whichthe changing includes updating, in response to receiving the inputvoice, the second avatar so as to change the display form of the secondavatar when voice is input to the voice input unit of the second userterminal with the line of sight of the second avatar being directedtoward the first avatar. The changing further includes displaying, whenthe second avatar exists within the visual field of the first avatar,the visual-field image including the updated second avatar on thehead-mounted device of the first user terminal.

(17) An information processing method according to Item (16), furtherincluding executing the changing of the display form of the secondavatar when the second avatar does not exist within the visual field ofthe first avatar.

(18) A system for executing the information processing method of any oneof Items (11) to (17).

What is claimed is:
 1. A method, comprising: generating first virtualspace data for defining a first virtual space, the first virtual spaceincludes: a first avatar, which is associated with a first userterminal; and a first virtual room, in which the first avatar and afirst virtual screen are arranged, wherein the first virtual screen isconfigured to display a moving image, and the first virtual room ismanaged by the first user terminal; detecting a movement of a headmounted device (HMD), wherein the HMD is included in the first userterminal; identifying a visual field within the first virtual space inaccordance with the detected movement of the HMD; displaying, on theHMD, a visual-field image corresponding to the visual field within thefirst virtual space; receiving, from a second user terminal, a visitingrequest signal for requesting that a second avatar associated with thesecond user terminal visit the first virtual room; updating the firstvirtual space data by updating a size of the first virtual screen basedon a total number of avatars arranged in the first virtual room;calculating billing information for both the first user terminal and thesecond user terminal based on the updated size; and updating thevisual-field image based on the updated first virtual space data.
 2. Themethod according to claim 1, further comprising updating the firstvirtual space data by updating a size of the first virtual room based onthe total number of avatars arranged in the first virtual room.
 3. Themethod according to claim 2, further comprising automatically enlargingthe size of the first virtual screen based on enlargement of the size ofthe first virtual room.
 4. The method according to claim 1, furthercomprising updating a size of the first virtual room so as to maintain arelative positional relationship between the first avatar and the firstvirtual room, wherein the relative positional relationship is apositional relationship exhibited prior to the updating of the size ofthe first virtual room.
 5. The method according to claim 1, furthercomprising: determining a billing amount to be billed to a first userfor viewing of the moving image on the first virtual screen; andupdating the billing amount to be billed to the first user based on theupdated size of the first virtual screen.
 6. The method according toclaim 5, further comprising: automatically enlarging the size of thefirst virtual screen based on enlargement of a size of the first virtualroom; and enlarging the size of the first virtual screen based on aninstruction issued by one of a first user or a second user, wherein theone of the first user or the second user is billed a first billingamount in response to automatic enlargement of the size of the firstvirtual screen to a predetermined size based on the enlargement of thesize of the first virtual room, the first billing amount being smallerthan a second billing amount to be billed to the one of the first useror the second user in response to the size of the first virtual screenbeing enlarged to the predetermined size based on the instruction issuedby the one of the first user and the second user.
 7. The methodaccording to claim 1, further comprising: receiving, from the first userterminal, an instruction for requesting that the size of the firstvirtual screen be enlarged to a predetermined size; and updating, basedon the received instruction, the first virtual screen so as to enlargethe size of the first virtual screen to the predetermined size.
 8. Themethod according to claim 7, further comprising: determining a billingamount to be billed to a first user for viewing of the moving imageperformed by the first user; and determining the billing amount to bebilled to the first user based on the received instruction.
 9. Themethod according to claim 1, wherein: the first virtual space furtherincludes the second avatar associated with the second user terminal anda third avatar associated with a third user terminal; and the methodfurther comprises: receiving, in response to a voice being input to avoice input unit of the second user terminal, voice data representingthe input voice from the second user terminal; generating a secondline-of-sight object based on a line of sight of the second avatar;generating a third line-of-sight object based on a line of sight of thethird avatar; updating the second line-of-sight object so as to change adisplay form of the second line-of-sight object; and displaying, inresponse to the updated second line-of-sight object and the thirdline-of-sight object existing within the visual field of the firstavatar, the visual-field image including the updated secondline-of-sight object and the third line-of-sight object on the HMD. 10.The method according to claim 9, further comprising: generating thesecond line-of-sight object based on the line of sight of the secondavatar in response to the second avatar existing outside of the visualfield of the first avatar; and generating the third line-of-sight objectbased on the line of sight of the third avatar in response to the thirdavatar existing outside of the visual field of the first avatar.
 11. Themethod according to claim 1, wherein: the first virtual space furtherincludes the second avatar associated with the second user terminal anda third avatar associated with a third user terminal; and the methodfurther comprises: receiving, in response to a voice being input to avoice input unit of the second user terminal, voice data representingthe input voice from the second user terminal; generating a secondline-of-sight object based on the received voice data and a line ofsight of the second avatar; and displaying, in response to the secondline-of-sight object existing within the visual field of the firstavatar, the visual-field image including the second line-of-sight objecton the HMD of the first user terminal.
 12. The method according to claim11, further comprising generating the second line-of-sight object basedon the line of sight of the second avatar in response to the secondavatar existing outside of the visual field of the first avatar.
 13. Themethod according to claim 1, wherein: the first virtual space furtherincludes the second avatar associated with the second user terminal anda third avatar associated with a third user terminal; and the methodfurther comprises: receiving, in response to a voice being input to avoice input unit of the second user terminal, voice data representingthe input voice from the second user terminal; changing a display formof the second avatar in response to the voice being input to the voiceinput unit of the second user terminal with a line of sight of thesecond avatar being directed toward the first avatar under a state inwhich the second avatar exists outside of the visual field of the firstavatar; and displaying, in response to the second avatar existing withinthe visual field of the first avatar, the visual-field image includingthe updated second avatar on the HMD.
 14. A system comprising: aprocessor; and a memory configured to store instructions thereon,wherein the processor is in communication with the memory, and theprocessor is configured to execute the instructions for: generatingfirst virtual space data for defining a first virtual space, the firstvirtual space includes: a first avatar, which is associated with a firstuser terminal; and a first virtual room, in which the first avatar and afirst virtual screen are arranged, wherein the first virtual screen isconfigured to display a moving image, and the first virtual room ismanaged by the first user terminal; receiving, from the first userterminal a detected movement of a head mounted device (HMD), wherein theHMD is included in the first user terminal; identifying a visual fieldwithin the first virtual space in accordance with the detected movementof the HMD; instructing the HMD to display a visual-field imagecorresponding to the visual field within the first virtual space;receiving, from a second user terminal, a visiting request signal forrequesting that a second avatar associated with the second user terminalvisit the first virtual room; updating the first virtual space data byupdating a size of the first virtual screen based on a total number ofavatars arranged in the first virtual room; calculating billinginformation for both the first user terminal and the second userterminal based on the updated size; and updating the visual-field imagebased on the updated first virtual space data.
 15. The system accordingto claim 14, wherein the processor is further configured to execute theinstructions for updating the first virtual space data by updating asize of the first virtual room based on the total number of avatarsarranged in the first virtual room.
 16. The system according to claim14, wherein the processor is further configured to execute theinstructions for updating a size of the first virtual room so as tomaintain a relative positional relationship between the first avatar andthe first virtual room, wherein the relative positional relationship isa positional relationship exhibited prior to the updating of the size ofthe first virtual room.
 17. The system according to claim 14, whereinthe processor is further configured to execute the instructions for:determining a billing amount to be billed to a first user for viewing ofthe moving image on the first virtual screen; and updating the billingamount to be billed to the first user based on the updated size of thefirst virtual screen.
 18. A system comprising: a first user terminal; asecond user terminal; and a server in communication with the first userterminal and the second user terminal, wherein the server comprises: aprocessor; and a memory configured to store instructions thereon,wherein the processor is in communication with the memory, and theprocessor is configured to execute the instructions for: generatingfirst virtual space data for defining a first virtual space, the firstvirtual space includes: a first avatar, which is associated with thefirst user terminal; and a first virtual room, in which the first avatarand a first virtual screen are arranged, wherein the first virtualscreen is configured to display a moving image, and the first virtualroom is managed by the first user terminal; receiving, from the firstuser terminal a detected movement of a head mounted device (HMD),wherein the HMD is included in the first user terminal; identifying avisual field within the first virtual space in accordance with thedetected movement of the HMD; instructing the HMD to display avisual-field image corresponding to the visual field within the firstvirtual space; receiving, from the second user terminal, a visitingrequest signal for requesting that a second avatar associated with thesecond user terminal visit the first virtual room; updating the firstvirtual space data by updating a size of the first virtual screen basedon a total number of avatars arranged in the first virtual room;calculating billing information for both the first user terminal and thesecond user terminal based on the updated size; and updating thevisual-field image based on the updated first virtual space data. 19.The system according to claim 18, wherein the processor is furtherconfigured to execute the instructions for updating a size of the firstvirtual room so as to maintain a relative positional relationshipbetween the first avatar and the first virtual room, wherein therelative positional relationship is a positional relationship exhibitedprior to the updating of the size of the first virtual room.
 20. Thesystem according to claim 18, wherein the processor is furtherconfigured to execute the instructions for: determining a billing amountto be billed to a first user for viewing of the moving image on thefirst virtual screen; and updating the billing amount to be billed tothe first user based on the updated size of the first virtual screen.